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APPARATUS FOR UNIFORM LINEAL MOTION BLUR ESTIMATION 
USING MULTIPLE EXPOSURES 

Technical Field of the Invention 

The present invention relates generally to estimating motion from image blur and, in 
5 particular, to an image sensor with a capability to capture multiple concurrent (overlapping in 
time) images which forms part of a system used to estimate a uniform lineal motion using 
error minimisation. 

Background Art 

Known arrangements for performing motion estimation of video images typically 
10 use one of the following methods: 

(i) estimation of illumination partial derivatives in combination with 

optical flow; 

(ii) local (block based) calculation of a cross-correlation between 
successive images and using maxima to estimate the shift between images; or 

15 (iii) estimation of the motion blur in any one image (using a variety 

of methods such as blind de-convolution, maximum likelihood estimation etc) to then use 
as a motion vector in optical flow. 

The above arrangements each suffer from the disadvantage of being time consuming 
and not necessarily very robust or reliable. For example, in video applications, at least two 
20 frame images are required to provide a single estimation of blur between the images. Further, 
since conventional video capture devices capture the frame images very quickly, and spend 
much of the time associated with the frame period outputting the image data, existing devices 
have no ability to estimate motion or blur within the frame (ie. the blur that occurred during 
the capture of the frame). «■ 
25 Motion estimation is typically used for video coding, object detection and tracking 

as well as video content analysis. 

Disclosure of the Invention 
It is an object of the present invention to substantially overcome, or at least 
ameliorate, one or more disadvantages of existing arrangements. 
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According to a first aspect of the invention, there is provided apparatus for 
estimating motion of a moving image, said apparatus comprising: 

an image sensor for capturing pixels of at least first and second images in which one 
of said images is sampled during a formation of at least one other of said images; 
5 means for comparing said images to determine at least one motion vector for 

association with one of said images. 

According to a second aspect of the invention, there is provided apparatus for 
estimating motion of a moving image, said apparatus comprising: 

an image sensor for capturing pixels of at least first and second images in which one 
10 of said images is sampled after sampling of at least one other of said images, the sampling of 
said one image occurring before an outputting from said sensor of image data of said other 
image; 

means for comparing said images to determine at least one motion vector for 
association with one of said images. 
15 According to a first aspect of the invention, there is provided apparatus for 

estimating motion of a moving image, said apparatus comprising: 

an image sensor for capturing pixels of at least first and second images characterised 
in that images data of each of said images is acquired prior to an outputting of said image 
data from said sensor; 

20 means for comparing said images to determine at least one motion vector for 

association with one of said images. 

Other aspects of the invention are also disclosed. 

Brief Description of the Drawings 
A number of preferred embodiments of the present invention will now be described 
25 with reference to the drawings, in which: 

Fig. 1 shows blur functions b x and b 2 representing lineal (motion) blur of spatial 
extent 2ai and 2a 2 respectively occurring in a direction at angle 0to the x axis; 

Figs. 2A and 2B illustrate the x' component of the respective blur functions bj and 
b2 , resulting from two uniform exposures with the same start times, but with the one 
30 exposure twice the duration of the other; 
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Figs 3 A and 3B illustrate an interpretation of error minimisation; 
Fig. 4 is a schematic block diagram of an apparatus for blur estimation; 
Fig. 5 is a schematic block diagram of a structure of a CMOS sensor used to capture 
blurred images g\ and g 2 of Fig. 4; 
5 Fig. 6 is a timing sequence to be used with the CMOS sensor of Fig. 5 to capture the 

two overlapping blurred images g\ and g 2 \ 

Fig. 7 is a schematic block diagram of an embodiment of a correlator that may be 
used in the apparatus of Fig. 4 for blur estimation; 

Fig. 8 is a schematic block diagram of another embodiment of a correlator that may 
10 be used in the apparatus of Fig. 4 for blur estimation; 

Fig. 9 is a schematic block diagram of an embodiment of a extreme locater that may 
be used in the apparatus of Fig. 4 for blur estimation; 

Figs. 10A to 10E provide comparisons between image capture according to prior art 
arrangements and those according to embodiments of the present invention; and 
15 Figs. 1 1 and 12 depict capture devices according to embodiments of the invention. 

Detailed Description including Best Mode 
Prior to the description of the preferred embodiments of this invention, motion 
estimation using blur analysis will first be discussed. 

Consider an ideal two dimensional digital image signal, represented as a function 
20 f(x,y), where x and y are the horizontal and vertical image coordinates. The image signal 
f(x,y) is defined for all integer values of x andy, and undefined for all non-integer values of 
x andy. The digital image signal f(x,y) is typically obtained by sampling an analog image, 
for instance, an image on film. 

In practice, this ideal image signal f(x 9 y) is distorted by various image degradations. In 
25 this specification, image blurring is considered as well as additive noise. The image blurring 
to be considered is a special form of blur, namely uniform lineal blur typically caused by 
motion in a plane at an oblique angle to a line of sight of a camera used for capturing the 
image. The blurred image is represented as g v In the preferred embodiment of the invention, 
two different blurred images are used, g x and g 2 . 
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The blurred images g { and g 2 can be represented as the result of a pair of blur functions 
b x and b 2 operating on the image signal f(x,y). The blurred images g l and g 2 can therefore 
be represented using the convolution (*) operator as: 

s gi(x,y)=f(x,y)*bi(x,y)+ni(x,y) (i) 

g2(x,y) =f(x,y)*b 2 (x,y) +n 2 (x,y) (2) 

where nj and n 2 are additive noise signals. 
10 An assumption is made that the motion is lineal and uniform over the image region 

under consideration. The blur functions b x and b 2 represent one dimensional (motion) blur of 
spatial extent 2a j and 2a 2 respectively occurring in a direction at angle 6 to the x axis as 
illustrated in Fig. 1. Consequently the blur functions b x and b 2 can be represented as follows: 

b x (x 9 y) = ^rect{-f-).5{y') (3) 
2a, 2a, 



15 



b 2 (x, y) = -l-rect(^-^).S(y' - y' Q ) (4) 



2a 2 2a~, 



wherein 

r 



red 



— I = 0 for p > s , or 
2s} 



1 for l/^s , and (5) 

20 S(p) is the Dirac delta function such that: 

[ f(x).S(x-x 0 )dx = f(x 0 ) (6) 

J- CO 

wherein /is any function. (jc',j/)are rotated coordinates having the following relations: 



x = +jt cos(#) + v sin(0) 1 
25 v y ^ v /i ^ 

y f = -x sin(0) + y cos(0) J 
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As will be apparent from Fig. 1, the rotated coordinates have one coordinate aligned 
with the direction of blur. The centre of the second blur function b 2 is offset by an amount 
(x 0 ,y 0 ) from the centre the first blur function b v In essence, the blur functions ft, and b 2 
5 convert all point intensities of the image signal f(x, y) into linear smears of length directly 
proportional to the exposure time and the image velocity. 

The two dimensional Fourier transform G(u,v) of an arbitrary signal g(x,y) is defined as: 

00 00 

G(u,v)= J jg(x,y)exp(-2m[ux + vy])dxdy (8) 

-oo-oo 

10 where u and v are the spatial frequency coordinates. Consequently, the transforms Gj and G2 
of the blurred images gj and g2 can be written as: 

Gi(u,v)=F(u,v).Bj(u,v) + Nj(u,v) (9) 

15 G 2 (u, v) =F(u, v).B 2 (u, v) +N 2 (u, v) (10) 

The discrete Fourier transforms Bj and B 2 of the blur functions bj and b 2 have a simple 

(one dimensional) form when expressed in relation to rotated frequency axes, namely: 

u' = +wcos(0) + vsin(0)j 
v' = -u sin(0) + vcos(#) J 



20 



^^a^ (12) 
2m , a l 



n , x sin(2;zw'a 0 ) , ^ r , , , x 

* 2 (k,v) = v , 2) .exp(-2m[u f x' 0 ]) (13) 
2ot a, 



*2 



From Equations (9), (10), (12) and (13), the transforms Gj and G 2 can be rewritten as: 

G, (w,v) = F(w, v).— — (14) 

2;zw a. 



25 
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G 2 (u,v) = F(u,vy 



sin(2^w'a 2 ) 
27iu'a-> 



(15) 



For simplification, the additive noise spectra terms Nj and N2 in Equations (9) and (10) 
have been substituted by a single noise spectrum term TV in Equation (15). 

Figs 2A and 2B illustrate the x' components of the respective blur functions bj and 62 
resulting from two uniform exposures, starting on the same time, but with the one exposure 
twice the duration of the other. This particular relationship between the two exposures results 
in the following parameters: 



a 2 = 2a 



y'o=o 

Applying these parameters to Equations (14) and (15) gives the Fourier spectra: 

sin(2mi'a) 



G l (w ? v) = F(w,v). 



27vu f a 



(16) 



(17) 



sm(4mi r a ) / . 

G 2 (w, v) = F(w,v). .exp(- 2mu'a)+ N{u, v) 

Ann* a 

From Equations (17) and (18), the noise term N(u,v) is as follows: 



(18) 



15 



N(u 9 v) = G 2 (u 9 v) - G l (u 9 v).cos(27m'a),oxp(~2mu f a) 



(19) 



20 



A modulus-squared error or "deviation" function t can be expressed as a function of the 
blur parameter a : 

t(d) = ^\N(u,v)\ 2 dudv 

= J jjG 2 (u,v) — G l (u,v).cos(2mi'a).exp(-2mu'a)\ dudv 



(20) 



However, Equation (20) can be further expanded as: 
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t(a) = H\g 2 \ 2 +|G,| 2 cos 2 (2mi'a)\ludv 

- j^{plG r exp(-2mu'a)+ GiG 2 .exp(2mu'a)).cos(27m'a)dudv 



(21) 



n 2 + 



dudv 



+ J 1^ |G, | cos(4;rw 'a)dudv 
-\\[{GlG x +G;G 2 )iudv 

~ ~ 1 1 ^' " ex P(~^ ;nM fl ) + ^2 • exp(4;riw 'a) jrfwrfv 



(22) 



5 Applying a transformation: 



to Equation (22) results in the following: 



acos# = X| 
asin0 = r 



(23) 



t(a,9) = e(X,Y) 



>dudv 



-\\l{G' 2 G i+ G;G 2 )dud V 

+ \ \ I \ G S cos(4a[uX + vY])dudv 

- - f^{G' 2 G v exp(-4m[uX + vY]) + G,*G 2 .exp(4m[uX + vY}]dudv 



■ (24) 



10 



Because the blurred images gj and g2 are both real functions, the following properties 



exist: 



G' n (u,v) = G„(-u ,-v) 
\G„{-u,-v) = \G n (u,v) 



(25) 
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The full (or at least symmetrical) limits in frequency space allow the last two terms of 
Equation (24) to be joined. 

e(X,Y) = 

££{ |G2(M ' v)|2+ i |Gi( " ,v)|2 W v 

~\ £ £fe(",v)G,(«,v) + G;(u,v)G 2 (u,v)}tudv. (26) 

+ \ £ £l Gl v ^| 2 ex P( 4 ^["^ + vYjjdudv 

- f f G,(-" ,-v)G 2 (u,v).exp(4m[uX + vY])dudv 

Only the last two terms of Equation (26) are functions of X and Y. These are therefore 
the only terms determining where the minimum of the deviation function s(X, Y) is located. In 
particular, the last two terms can be recognised as having the form of auto and cross- 
correlation functions. 

- f f l G i ( w > v >| 2 exp(4^r[wZ + vY^dudv 

- £ £ G, (- u,-v)G 2 (u, v). e\p(4m[uX + vY])dudv . (27) 
= - f f G i (~ u " v )( G i (" u ~ v ) - 2G 2 ("> v )}- exp(4OTLx" + vY\ludv 

Equation (27) can be interpreted as the convolution of gi(-x,-y) with the difference of 
gi(-x f -y) and 2g2(x,y) evaluated at coordinates (X\ Y') , where 

\ (28) 
Y' = 27 J V ' 

Blur functions having such a result are illustrated in Figs. 3A and 3B. 
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It follows from the foregoing that where two images are captured in direct association, 
either overlapping, immediately adjacent or closely adjacent with respect to capture time, and 
assuming linear motion over each area of interest, the blur in either image may be 
determined. With conventional arrangements this requires two image captures, generally 

5 separated by one field or frame period, to determine motion vectors. 

The present inventors have realised that where more than one image can be captured 
with a field or frame period, motion blur may therefore be estimated for at least that field or 
frame. Thus motion vectors may be obtained from multiple sub-field or sub-frame images, 
thereby improving motion vector accuracy and associating such vectors with a single 

10 (traditional) field or frame image, as opposed to a pair of such traditional images. In this 
fashion the motion vector obtained much more accurately represents the motion that occurred 
during the capture of the respective image, as opposed to the motion that occurred between 
capture of adjacent traditional field/frame images. 

Fig. 4 illustrates a block diagram of apparatus 9 for estimating the blur parameters of 

15 blurred images according to an embodiment of the present invention. The apparatus 9 has an 
image sensor 10, preferably formed in CMOS technology, for capturing the blurred images gi 
and g2, a plurality of correlators 20 and 30 for performing autocorrelation of the blurred 
image g\ and cross-correlation between the two images gj and g 2 respectively, an error 
function calculator 40 for evaluating an error function t over all possible displacements using 

20 the results from the correlators 20 and 30, and an extreme locater 50 for finding the 
displacement with the minimum value for the error function 

The operation of the image sensor 10, as implemented with CMOS technology, will be 
further explained with references to Figs. 5 and 6. Fig. 5 shows a circuit layout of a part 12 
of the image sensor 10, the part 12 being arranged to capture a single pixel of the digital 

25 image signal f(x f y) by means of a pixel cell 60. 

The pixel cell 60 is formed of a photodiode 63 which acts as an image pixel sensor and 
which connects to a transistor 64 to discharge a parasitic capacitor Cp according to the 
intensity of illumination falling upon the photodiode 63. The parasitic capacitor Cp may be 
formed by a floating diffusion. A transistor switch 62, operated by a reset signal 'res', acts to 

30 reset the capacitor Cp by applying a supply rail potential to the capacitor Cp. A signal "TX" 
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is applied to the parasitic capacitor Cp enabling the capacitor Cp to discharge through the 
operation of the phptodiode 63. It will be appreciated therefore that the sensor 10 operates 
according to negative logic principles where the charge on the capacitor Cp is the inverse of 
the light intensity-duration function applied to the photodiode 63. A select signal 'sel' 
5 operates a transistor switch 65a which, together with a source- follower transistor 65b, couples 
the voltage on the parasitic capacitor Cp to a pixel data bus 6 1 . The duration that the signal 
*TX' is applied is usually called the exposure time, and the charge of the parasitic capacitor 
Cp is the value of the pixel cell 60. A number of the pixel cells 60 may be connected to the 
pixel data bus 61. 

10 In order to provide a sequence of output pixel values of an image, a switching 

arrangement 66 is provided to couple such values to an output 67 of the part 12. The 

sequence may, for example, be a raster sequence. 

Signals "Nl" and "N2" are applied to transistor switches SW1 and SW3, causing them 

to conduct thereby providing for noise charge values to be stored on capacitors CI and C3 
15 respectively. Such noise values are representative of circuit noise and are available on the 

pixel data bus 61 when the 'seF signal of each of the cells 60 connected to the bus 61 are all 

disabled or not selected. 

Signals 'SI' and 'S2' are applied to turn ON switches SW0 or SW2 respectively, so that 

capacitors CO and C2 are selectively charged according to the pixel value as present on the 
20 pixel bus 61. 

Signals 'HSTO'and 'HST1" are applied to turn ON switch pairs SW4 and SW5, and 
SW6 and SW7 respectively. The switches SW5 and SW7 act to couple the noise values on 
the capacitors CI and C3 respectively to a buffer 'bufh', while the pixel values present on the 
capacitors CO and C2 are coupled via operation of the switches SW4 and SW6 respectively to 
25 a buffer 'bufs'. A differential amplifier 'diffamp', connected to the buffers 'bufh' and 'bufs', 
finds the difference between the two signals to output a signal indicative of the light fallen 
upon the photodiode 63. 

By forming the image sensor 10 as a matrix of pixel cells 60 formed in rows (lines) and 
columns, overlapped double sampling of an image that gives rise to blur functions illustrated 
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in Figs. 2 A and 2B, can be achieved using the circuit 12 of Fig. 5 by applying to the circuit 12 
a signal sequence as shown in Fig. 6. 

When a signal c selO' is asserted, all the pixel cells 60 in row 0 of the matrix are enabled, 
and each drive a corresponding pixel data bus 61 in the corresponding column. 
5 The signal 'res' is then asserted, causing all the pixel cells 60 in line 0 to be reset, and 

leaving each cell 60 driving a noise value on the corresponding pixel data bus 61. 

As the signal 'res' id de-asserted, the signals 'Nl' and 'N2' are then asserted to latch the 
noise values into capacitors CI and C3. The signal *TX' is then asserted for a period 'Tex'. 
During the period Tex', all the pixel cells 60 in line 0 discharge their corresponding parasitic 
10 capacitors Cp according to the light intensity on line 0. The signal 'ST is asserted in the 
middle of the exposure time to store the pixel values at that time in the respective capacitors 
CO. At the end of the exposure time 'Tex', the pixel values at that time are stored in 
capacitors C2 by asserting signal 'S2'. The signal 'HST0' is then asserted to switch on SW4 
and SW5 to find the true values of the pixels during the first half of exposure time 'Tex'. 
15 The signal 'HST1 ' is asserted to switch on SW6 and SW7 to find the true values of the pixels 
during the second half of exposure time 'Tex'. The signals 'HST0' and 'HST1' are asserted 
progressively along line 0 for each pixel to output a sequence of pixel values for the line. 
This is repeated until all the pixels in line 0 are read. 

Such a mode of operation results in a pixel sequence as follows, for n pixels in the line: 
20 Pixel_l_half, PixeMJull, Pixel_2_half, Pixel_2_full, . . . Pixel_n_full. 

A next line (line 1) is selected by asserting 'sell', and the above is repeated. 
Alternatively, the mode of operation may be altered to provide the following sequence: 
Pixel_l_half, Pixel-2_half ... Pixel_n_half, Pixel l_full, Pixel 2JU11, ... 
Pixel_n_fiill. 

25 It will be apparent from the forgoing that where SI is activated at the middle of TX and 

S2 at the end, the first image obtained by virtue of SI, for a stationary target which presents 
no blur, will have half the intensity of the second image obtained at S2. As such, any image 
derived at SI will include, for all intents and purposes, the same components of the image 
obtained at S2. This is to be contrasted with prior art arrangements where single frames or 
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fields are captured one after the other in which case the entire sensor reset and newly 
illuminated. 

Fig. 7 shows one embodiment of a circuit 69 for use as part of the correlator 20 or the 
correlator 30. In this embodiment, the circuit 69 has registers 70 to 73 for holding the pixel 
data of a first image p to be correlated and one cyclic barrel shifter 74 to 77 for each row. 
The cyclic barrel shifters 74 to 77 rotate the pixel data to perform the convolution. The 
circuit 69 also has one cyclic barrel shifter 78 to 81 for each column to rotate the pixel data 
from the row barrel shifters 74 to 77. The barrel shifters 78 to 81 output to corresponding 
groups of multipliers 82 for multiplying the rearranged pixel data with pixel data from 
another image source q, and an adder 83 for adding the products from the multipliers 82 
together. When the circuit 69 is used as the auto-correlator 20 of Fig. 4, the input images p 
and q are derived from the same input image source. Although not shown in Fig. 7, the q 
input may be deposited into registers in a manner akin to the p input, thereby enabling 
correction for any timing issues. 

The circuit 69 commences operation by shifting into the registers 70 to 73 the pixel data 
from the image /?. After the whole image p is shifted in, the barrel shifters 74 to 77 are 
configured to rearrange the pixel data of image p to calculate various correlation values. In 
the situation where the barrel shifters 74 to 77 are not shifting its input p, the correlator 
calculates (p o <7)(0,0) . With the row barrel shifters 74 to 77 shifted one position to the right, 
and the column barrel shifters 78 to 81 not shifted, the circuit 69 calculates (p o q)(Q,l) , 
which is the same as (p o q)(0, N — 1) , with TV being the number of rows in the images p and 
q. When the row shifters 74 to 77 are not shifting their input p, and the column shifters 78 to 
81 are shifting their input q by k positions to the right, then the circuit 69 calculates both 
(p o q)(k,0) and (p o q)(M - k,0) , with M being the number of columns in the images p and 

In terms of hardware, for images of dimension MxN pixel cells 60, this embodiment of 
the circuit 69 requires: 

- 2xMxN registers to hold all the pixel values of the two images p and q\ 

- M TV-input row barrel shifters to shift the pixel values in the rows; 
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- N M-input column barrel shifters to shift the outputs from the row barrel 

shifters; 

- MxN multipliers to multiply the rearranged pixel values of one image p with 

pixel values from the other image q\ 

- Mx(N-\) 2-input adders (or a single MxN input adder) to add the products of the 



multipliers; and 
- barrel shifter control logic. 
Fig. 8 shows a second embodiment of a correlator circuit 85. This embodiment uses 
Fourier Transforms to calculate correlations between images c and d. Using the correlation 
10 theorem, 



the cross-correlation of the two images c and d can be calculated by inputting pixels from the 
images c and d into respective row buffers 86. A Fourier transform 87 is then performed on 

15 both images c and d to obtain their Fourier coefficients C and D. The coefficients are 
multiplied 88 on a one by one basis and an inverse Fourier transform 89 is performed on the 
output of the multiplier 88. The results of the inverse transform 89 comprise cross 
correlation coefficients that are retained in a result buffer. Again, for an autocorrelator 
implementation, the inputs c and d are connected to the same image input. 

20 The implementation of the 1 -dimensional Fourier transform and inverse Fourier 

transform blocks are well known to persons in the art. 

In terms of hardware, to correlate two images c and d of dimension MxN, the 
arrangement 85 of Fig. 8 requires: 

- two TV-pixel buffers to store one row of the incoming data; 

25 - two blocks of logic performing 1 -dimensional Fourier transform on the row of 



y) o d (x, y) o C* (w, v)D(u, v) 



(29) 



//pixels. The amount of logic required may be determined from 
the value of N\ 



- two blocks of logic performing 1 -dimensional Fourier transform on the row of 



30 



M coefficients. The amount of logic required may be determined 
from the value of M\ 
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- three transposing memories to hold MxN pixel values/coefficients; 

- three memory blocks to hold MxN Fourier coefficients, two for the input of the 

multiplier and one for the product; 

- one block of logic performing 1 -dimensional inverse Fourier transform on M 
5 coefficients; 

- one block of logic performing 1 -dimensional inverse Fourier transform on TV 

coefficients; and 

- one //-entry buffer to hold one row of the correlation coefficients. 

The error function calculator 40 of Fig. 4 receives the correlation results from the 
10 correlators 20 and 30 and calculates the error values based on the foregoing equations derived 

above. The error calculator 40 may be formed in hardware by an appropriate combination of 

adders and multipliers and associated logic, or alternatively implemented in software. 

The evaluation of the error function for every point in the domain of the correlation 

function provides identification of two minima in the error function, one opposite to the other 
15 with respect to the origin. Either one of these minima can indicate the displacement (blur) 

between the two images. To find these minima, all the error function values are searched for 

the smallest error function value, and the displacement that result in the minima are recorded. 

This takes MxN clock cycles, where M is the number of rows (lines) in the image and TV is the 

number of columns in the image. 
20 However, to increase the accuracy in the estimation, the true minimum in sub-pixel 

precision can be found. A typical method used is curve fitting, which is a method well 

known to people skilled in the art. 

Before the implementation of such an extreme locater is considered, the theory behind 

the extreme locator will be examined. Consider for example a (2«+l)x(2«+l) neighbourhood 
25 around pixel (x,y) wherein the neighbourhood is defined by all the pixels in a square with 

corners at (x-n 9 y-n) 9 (x-n,y+n), (x+n,y+n) and (x+n,y-n). Typically n~l or 2. A quadratic 

surface can be fitted locally around the minimum, after which a location of a best-fit surface 

minimum is calculated. This is effectively interpolation of the minimum position. 
A second order surface can be expressed as: 

30 
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P{x,y) = a oa + a i0 x + a 0l y + a u xy + a 20 x 2 + a 02 y 2 



(29) 



which represents a local region with an elliptical paraboloid of arbitrary orientation and axes. 

A 'least squares fit' is applied to the function f(x 9 y) on a discrete grid of points: 

x = mA 
y = nA 

expressed as: 

E =H *Z[f{mA,nA)-P(mA,nAj\ 2 

n=-Nm=-M 



(30) 



(31) 



Stationary points for all values of k and 1 are found by: 



dE = I fl(»A)*(iiA) , [/(ifiA,iiA)-i J (ifiA > «A)] = 0 



da 



kl n=-Nm--M 



(32) 



Six simultaneous equations can be derived. Typically M=N=l or 2. 



I&v/^x^v^ 



(33) 



Equation (33) is expanded into: 
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(34) 
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It is noted that the square matrix in Equation (34) can be greatly simplified if the chosen 
polynomials are orthogonal over the discrete range chosen. However, in this case they are not 
quite orthogonal, but all odd summations do cancel out. Therefore: 
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(35) 



Equation (35) is further simplified using the known relationships: 



10 and 



A = B.C 



B \ A = C 



(36) 
(37) 



15 



The dominant diagonal in matrix B simplifies the calculation of the matrix inversion B" 1 . 
Differentiation of equation (29) results in: 



dP 

— = a lQ +a u y + 2a 20 x = 0 
ox 



dP 
dy 



= a 0] + a u x + 2a 02 y = 0 



(38) 



Equation (38) can explicitly be rewritten for the location of the extreme 
CWoW"^^*) as: 



20 
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x 0 = fl '° (2fl02 " fll ' ) and y 0 = < la ^ a ^ (39 ) 
a u -4a 10 a 02 a u -Aa 20 a ai 



For M=N=l, the components of matrix B are: 

n=-\m=-l 

n=-lm=-l n=-lm=-1 
n--\m--\ 

tt» 4 = ZZ» 4 =« 



fl=-lm=-l /t=-l/n=-l 



Similarly, for M=iV=2 the components of matrix B are calculated as: 



2 2 



ZZl = 25 

Fj=-2m=-2 

zi> 2 =zi> 2 =*> 

n=-2m=-2 n=-2m=-2 

XZ«W = ioo 

n=-2m=-2 
2 2 2 2 



n=-2m=-2 n=-2m=-2 



(40) 



(41) 



10 The linear summations of the sampled function (matrix A in Equation (36)) can now be 
calculated. 
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(42) 



Explicitly, Equation (35) can be written as follows for the 3x3 neighbourhood (M=7V==1): 
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(43) 



and matrix inversion of Equation (43) gives: 
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10 



15 



20 



Given the foregoing, curve fitting may be used to locate the extremes of the error 
function. An embodiment of an implementation of an extreme locater 50 operating using the 
curve fitting method is shown in Fig. 9. 

A minimum neighbourhood locater 97 finds the smallest one of the error coefficients 
output from the error function calculator 40. The minimum neighbourhood locater 90 further 
stores the coefficients in a number of line stores 91 (which preferably holds between 3 and 5 
lines). When a new minimum is located, the locator 97 writes the 3x3 or 5x5 coefficients 
surrounding the minimum to a Minimum Neighbourhood buffer 92. 

The buffer 92 outputs to an image moment calculator 93 which calculates the following 
terms: 

^ j ^ 1 fm,n 
n m 

n m 

n m 

n m 

n m 

\ n m 

which are seen as one primary component of Equation (44). 

A matrix multiplier 94 multiplies the terms of equation (45) with the known matrix 
coefficients derived from Equation (40) or (41), and which are stored in a matrix coefficients 
memory 95. The coefficients aoo, aio> aoi, a2o, an, and ao2 can then be calculated using 
Equation (44). 

A displacement calculator 96 accepts the outputs of the matrix multiplier 94 and 
implements Equation (39) (reproduced below) using adders, multipliers and dividers to 
implement the desired functions: 



(45) 



a io( 2a 02- Q ll) md y = a 0l( 2fl 2Q- fl ll) 



a u -4a 20 a 02 



(39) 



a u -4a 20 a 02 

The outputs of the displacement calculator 96 are the horizontal and vertical 
displacements (x,,,)^) of the input images gi and g2 of Fig. 4 in relation to each other. The 
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horizontal and vertical displacements (x 0 ,y 0 ) are then supplied to subsequent devices as a 
motion vector of the block of images g/ and g 2 - The motion vector may be applied or utilized 
as required to perform a variety of functions, such as those traditionally associated with 
motion vectors. 

Accordingly, by utilizing sub-frame or sub-field sampling of an image through the 
arrangement of Fig. 5, it is thus possible to obtain a motion vector from the equivalent of 
single prior art image sample, whereas according to prior art configurations, at least two 
traditional samples would be required. Thus the described embodiment provides for an 
estimation of motion blur for each traditional image capture, be it a frame or an interlaced 
field. 

The above can be readily seen from Figs. 10A to 10E. Fig. 10A shows a prior art video 
signal 100 where an image frame is captured during period 102 and output from the sensor 
during a period 104 prior to the next frame 102 being captured. Fig. 10B shows a similar 
signal 106 for an interlaced frame where a first field 108 is captured and output over a period 
1 10. A second field 1 12 is captured and output over a period 1 14 prior to the first field of the 
next frame being captured. 

According to an embodiment of the invention, as shown in Fig. 10C, a first image 122 is 
captured followed by a second image 124, the second image 124 for example corresponding 
to the first field 108 of Fig. 10B. The data of the two images 122 and 124 is output during a 
period 126 before images 128 and 130 are captured for the next field and output during the 
period 132. 

Whilst Fig. 10C shows the motion blur images being captured one immediately after the 
other, such need not always be the case. As seen in Fig. 10D, a capture sequence 134 
includes the capture of a first blur image 136 overlapping with the capture of a second blur 
image 138. In Fig. 10E, a capture sequence 140 involves capturing one image 142 over a 
period twice that of another image 144. 

The multiple capture ability of the described arrangement derives from the duplication of 
the switching arrangements 66 of Fig.5. As a consequence, further duplication can result in 
further images being captured in association. Such may be useful in high speed photography 
of transient events where the capture time permits capturing the event, but where the time 
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required to output buffer the image data exceeds the time available to enable use of 
traditional video systems. 

A specific advantage afforded by the present invention is that the image sensor 10 used 
to capture the images from which motion blur is estimated, can be implemented in a number 

5 of configurations. In one configuration, shown in Fig. 11, the image sensor 10 can be the 
sensor from which the image to be ultimately displayed or otherwise reproduced is obtained. 
There, a video camera 150 is illustrated which includes a lens 152 and a charge-coupled 
device sensor 154 constructed according to the arrangement of Fig. 5. The sensor 154 
outputs the two blur images g } and g 2 to a motion blur estimation unit 156 according to the 

10 above description. The image g 2 also forms an output image of the camera 150 and is 
combined in a summer with motion vector meta-data 164 output from the unit 156. A delay 
158 such as a frame store may be used to ensure that the motion blur data is synchronised to 
the corresponding output image frame. The combined signal may be stored on a recording 
medium, such as tape or disk, for output 168 from the camera 150. 

15 Alternatively, as shown in the digital video camera 170 of Fig. 12, the image sensor 10 

can be additional to that used to form the image output. In Fig. 12, the camera 170 includes a 
traditional lens 180 and CCD sensor 182 for capturing images in a traditional fashion. A 
further image sensor 174 constructed according to the principles of Fig. 5 is provided and 
associated with a corresponding lens 172 for detection of image motion blur. Where desired, 

20 the camera 170 may include only a single lens (not illustrated) and associated optics for 
directing the image being viewed to each of the sensors 174 and 182. As seen from Fig. 12, 
the sensor 174 is depicted smaller than the sensor 182. In this regard, since motion blur is 
generally regional within an image, rather than randomised on a pixel-by-pixel basis, a lower 
resolution sensor may be used. Such facilitates optimising the computational complexity of 

25 the motion estimation unit 176 by simplifying the correlations and transforms where 
appropriate. A synchronisation unit 184 is provided to ensure the appropriate motion vector 
meta-data 178 is matched with the corresponding image output from the sensor 182. Other 
components of the camera 170 perform corresponding functions to those of Fig. 11. 
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The foregoing describes only some embodiments of the present invention, and 
modifications and/or changes can be made thereto without departing from the scope and spirit 
of the invention, the embodiments being illustrative and not restrictive. 
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QajjBs; iha claims defining the invuHiK,n uie *~> ;oiiow^ 



1 A. Apparatus for estimating motion of a moving image, said apparatus comprising: 

an image sensor for capturing pixels of at least first and second images in which one 
5 of said images is sampled during a formation of at least one other of said images; 

means for comparing said images to determine at least one motion vector for 
association with one of said images. 

IB. Apparatus for estimating motion of a moving image, said apparatus comprising: 
10 an image sensor for capturing pixels of at least first and second images in which one 

of said images is sampled after sampling of at least one other of said images, the sampling of 
said one image occurring before an outputting from said sensor of image data of said other 
image; 

means for comparing said images to determine at least one motion vector for 
15 association with one of said images. 



1C. Apparatus for estimating motion of a moving image, said apparatus comprising: 

an image sensor for capturing pixels of at least first and second images characterised 

in that images data of each of said images is acquired prior to an outputting of said image 
20 data from said sensor; 

means for comparing said images to determine at least one motion vector for 

association with one of said images. 

2. Apparatus according to claim 1A, IB or 1C wherein said image sensor comprises, 
25 for each said pixel: 

an pixel sensor; and 

pixel sampling means for sampling a value of said pixel sensor at a time 
corresponding to each of said images. 
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3. Apparatus according to claim 2 wherein said image sensor further comprises, for 

each said pixel, reset means for resetting the corresponding said pixel sensor after sampling 
of each of said images. 

5 4. Apparatus according to claim 3 wherein said image sensor comprises a pixel bus 

configured for sequential coupling to a plurality of said pixel sampling means, and a 
switching arrangement associated with said pixel bus, said switching arrangement comprising 
a least first and second storage devices corresponding to each of said images and each being 
configured to store temporal value of said pixel sensor at said corresponding time, and 
10 switching means for outputting said temporal values to represent corresponding pixel values 
of said images of said images. 

5. Apparatus according to claim 4 wherein said pixel sensor comprises a cell capacitor, 
a voltage upon said cell capacitor being modified by exposure of said pixel sensor to an 

15 image to be captured, and said storage devices comprise storage capacitors. 

6. Apparatus according to claim 4 wherein said image sensor comprises plural said 
pixel buses and associated switching arrangements, each associated with plural of said pixel 
sensors and corresponding pixel sampling means thereby forming a matrix of said pixel 

20 sensors. 

7. Apparatus according to claim 1A, IB or 1C, wherein at least said second image 
comprises an exposure duration greater than that of said first image, thereby providing at least 
said second image as blurred representation of said first image where a target of said images 

25 presents motion. 

8. Apparatus according to claim 7 wherein said second image is a predetermined 
integer multiple of an exposure duration of said first image. 
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9. Apparatus according to claim 7, wherein exposure durations of said first and second 
images overlap. 

10. Apparatus according to claim 1A, IB or lC,wherein said means for comparing 
5 comprises: 

an auto-correlator for calculating auto-correlation of said first image; and 
a cross-correlator for calculating cross-correlation between said first image and said 
second image; and 

error means for minimising an error function between said auto-correlation and said 
10 cross-correlation. 

11. An image sensor according to claim 10 wherein said error means comprises an error 
function calculator for determining an error function value at each pixel location between 
said auto-correlation and said cross-correlation, and an extreme locator for thresholding said 

15 error function value at each said pixel location to provide movement parameters 
corresponding to each said pixel location. 

12. An image sensor according to claim 1 1 wherein said movement parameters comprise 
direction and magnitude. 

20 

13. An image sensor for capturing pixels of at least first and second images 
characterised in that at least one of said images is sampled during a formation of at least one 
other of said images; 

25 14. An image sensor according to claim 13 wherein said at least two captured images 
have each have a corresponding start exposure time and different end exposure times. 

15. An image sensor according to claim 14, said image sensor comprising: 

an array of pixel cells, each said pixel cell for storing an electrical charge equivalent 
30 of an intensity of light to which said cell is exposed; and 
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at least two arrays of capacitors, each of said arrays of capacitors comprises capacitors 
electrically connected with corresponding pixel cells; 

wherein at each of said end exposure times said array of capacitors is arranged to store 
the charge of said array of pixel cells at said end exposure time. 

5 

16. Apparatus for capturing a moving image, said apparatus comprising: 

at least one image sensor according to any one of the preceding claims; 

means for controlling said image sensor to capture a sequence of said first and 

second images and for determining a motion vector relating to image blur between each pair 
10 of said first and second images in said sequence; 

means for combining the corresponding said motion vector with at least one of said 

first and second images in said sequence to output a sequence of images each having an 

associated motion vector. 

15 17. Apparatus according to claim 16 wherein said sequence of images output from said 
apparatus comprise a sequence of said second images. 

18. Apparatus for capturing a moving image, said apparatus comprising: 

a first image sensor, said first image sensor being configured according to any one of 
20 claims 1A to 15; 

means for controlling said first image sensor to capture a first sequence of said first 
and second images and for determining a motion vector relating to image blur between each 
pair of said first and second images in said sequence; 

a second image sensor for capturing a second sequence of images to be output from 
25 said apparatus; 

means for combining each said motion vector of said first sequence with a 
corresponding image in said second sequence to output from said apparatus a sequence of 
images each having an associated motion vector. 



459925 DBIP01 CFP1 564AU 



[I:\ELEC\CISRA\DBIP\Dbip01 ]459925.doc:ldp 



-27- 

19. Apparatus according to claim 1 8 wherein said images captured by said second sensor 
have a resolution exceeding that of those captured by said first sensor. 

20. Apparatus according to claim 18 or 19, said apparatus comprising optical means for 
5 viewing a scene to be captured and presenting said scene to each of said sensors. 

21. Apparatus according to any one of claims 16 to 20 further comprising 
synchronisation means for synchronising said motion vectors with corresponding images 
forming said output sequence. 

10 

.22. Apparatus for estimating motion of an image from a first and a second blurred 
image, said apparatus being substantially as described herein with reference to any one of the 
embodiments as that embodiment is illustrated in the accompanying drawings. 

15 23. An image sensor for capturing at least two captured images of an image, said image 
sensor being substantially as described herein with reference to Figs 5 and 6 of the 
accompanying drawings. 



20 DATED this FOURTEENTH Day of DECEMBER 1999 

Canon Kabushiki Kaisha 

Patent Attorneys for the Applicant/Nominated Person 
SPRUSON & FERGUSON 
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Fig. 5 
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